Repository navigation
Conversation
… conversion Replace the 400/100/4/1-year cycle decomposition and the month estimate-and-correct step in ord_to_ymd() (Modules/_datetimemodule.c) and _ord2ymd() (Lib/_pydatetime.py) with Figure 12 of Neri and Schneider, "Euclidean affine functions and their application to calendar algorithms" (Softw Pract Exper. 2023;53(4):937-970), which GCC's libstdc++, the Linux kernel, .NET and Go already use. The C version computes the year and the month steps with one multiplication each, and does the century step in 64 bits so that every positive int ordinal gives the same result as before. The pure-Python version uses plain floor division. Both versions return the same results as the old code for every ordinal from 1 to 3,652,059; the C version also for every positive int, and the pure-Python one for 2.3 million other integers, including negative and huge ones. In microbenchmarks on x86-64 (gcc 13, -O3), date.fromordinal() on random dates is about 1.4x faster, date + timedelta across a month boundary and date.fromisocalendar() about 1.2x faster, and the pure-Python date.fromordinal() about 1.3x faster. Add a test for the first and last days of every month in years at the start and the end of the 4-, 100- and 400-year cycles. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
| * ceil(2**32 / DI4Y), so the high 32 bits are the quotient and the low | ||
| * 32 bits encode the remainder. */ | ||
| const uint32_t year_scaled = 4 * day_of_century + 3; | ||
| const uint64_t year_product = (uint64_t)2939745 * year_scaled; |
There was a problem hiding this comment.
I don't like these magic numbers, can you please benchmark the simpler approach (compilers are pretty smart, and should be able to deal with these), like you implemented in Python? If we accept this (I still have to think about this), they should be as close as possible for maintenance purposes.
There was a problem hiding this comment.
Ok. I rewrote it to follow _ord2ymd() step by step, with plain / and % by constants. gcc 14 and clang 19 turn them into multiplications anyway (no div instructions at -O3).
Results, C implementation, µs per 1000 operations (pyperf, gcc 14.2, -O3, no PGO):
| Benchmark | main | magic numbers | simple |
|---|---|---|---|
date.fromordinal, random dates |
101 | 60.8 (1.65x) | 62.2 (1.62x) |
date + timedelta(days=40) |
104 | 82.0 (1.27x) | 85.2 (1.22x) |
date.fromisocalendar |
184 | 146 (1.26x) | 156 (1.18x) |
datetime + timedelta(days=40) |
102 | 78.7 (1.29x) | 83.2 (1.22x) |
date.fromordinal, consecutive dates |
65.9 | 61.9 (1.06x) | 59.8 (1.10x) |
| Geometric mean (6 benchmarks) | 1.25x | 1.21x |
Simple vs. magic numbers: 4 of 6 benchmarks not significant, geometric mean 1.03x slower. On its own, the function takes about 6.5 ns instead of 5.5 ns (the old code: about 17 ns). That isn't worth the magic numbers, so I've pushed the simple version.
It still matches the old code for every positive int ordinal (gcc and clang) and for all dates in the supported range.
| Speed up conversion of ordinals to dates in :mod:`datetime`, used by | ||
| :meth:`~datetime.date.fromordinal`, :meth:`~datetime.date.fromisocalendar` | ||
| and :class:`~datetime.timedelta` arithmetic, by switching to the | ||
| Neri–Schneider algorithm. |
There was a problem hiding this comment.
Add a link, and a rough % speedup.
Use plain division and modulo by constants instead of hand-derived multiply-shift constants; compilers emit multiplications anyway. Add a link to the paper and rough speedups to the NEWS entry. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
CC @cassioneri this may be of interest. |
|
|
||
| *day = n + 1; | ||
| /* Days since 0000-03-01, the start of the computational calendar. */ | ||
| const uint64_t days = (uint64_t)ordinal + 305; |
There was a problem hiding this comment.
I think we can drop the 64-bit arithmetic here entirely. As far as I can tell, the only reason for it is so that date.fromordinal() with an ordinal past MAXORDINAL still gets to the same "year must be in 1..9999, not N" error that it raises on main, because fromordinal only checks ordinal < 1 before calling ord_to_ymd. The other two callers never pass anything larger than MAXORDINAL + 2: normalize_date already range-checks the ordinal, and iso_to_ymd can reach at most 10000-01-02, the last day of ISO year 9999, which its caller then rejects.
Since the calendar repeats exactly every 400 years, fromordinal can handle the error case by shifting the ordinal into the first 400-year cycle and adding the years back afterwards, which keeps the error message identical without needing a 64-bit path in the hot function:
if (ordinal < 1) {
PyErr_SetString(PyExc_ValueError, "ordinal must be >= 1");
return NULL;
}
if (ordinal <= MAXORDINAL) {
ord_to_ymd(ordinal, &year, &month, &day);
}
else {
/* Out of range: find the year anyway, so new_date_subclass_ex()
* raises the usual "year must be in 1..9999, not N" error. The
* calendar repeats every 400 years (DI400Y days), so convert the
* equivalent date in the first 400-year cycle and add the years
* back. */
int cycles = (ordinal - 1) / DI400Y;
ord_to_ymd(ordinal - cycles * DI400Y, &year, &month, &day);
year += 400 * cycles;
}
return new_date_subclass_ex(year, month, day, type);With that change, every intermediate value in ord_to_ymd fits in a uint32_t — 4 * days + 3 tops out around 14.6 million — so the function can assert ordinal <= MAXORDINAL + 2 and the comment explaining why the century step is done in 64 bits can go away. I checked this version against the ord_to_ymd on main for every positive int, and the year, month and day all match.
I did not see a measurable speed difference on x86-64 with either gcc or clang, so this is mostly an argument for simplicity. It may help somewhat on 32-bit platforms, where the 64-bit division can turn into a library call, though I have not measured that. The pure Python version does not need an equivalent change, since it uses arbitrary-precision integers and floor division.
There was a problem hiding this comment.
Done, I used your version. ord_to_ymd() is now uint32_t only, with assert(ordinal <= MAXORDINAL + 2), and fromordinal() handles bigger ordinals with the 400-year shift. I checked it against main for all positive int, and the error message is the same. I also added a test that the error still has the correct year for some ordinals after MAXORDINAL.
I checked the changes twice.
* ord_to_ymd() uses only uint32_t; fromordinal() shifts larger ordinals by 400-year cycles, so the error message is unchanged. * Replace the hand-picked loop with a Hypothesis round-trip test and add tests for the error year after MAXORDINAL. * Explain each step in the comments of both implementations; cite the paper by DOI. * Update the NEWS entry with new measurements. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
test.test___all__ imports test.datetimetester, which now imports Hypothesis. Hypothesis imports the deprecated typing.AnyStr, and test___all__ fails on the unexpected DeprecationWarning. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
|
||
| try: | ||
| import hypothesis | ||
| with warnings.catch_warnings(): |
There was a problem hiding this comment.
I'm not really a fan of this, we're working around a Python 3.16 incompatibility in Hypothesis, and this is unrelated to the change here. The property test also doesn't cover much over the examples, also most Buildbots don't have Hypothesis installed so only the @examples run IIRC. I'd prefer reverting to the previous version of the test and dropping the Hypothesis changes. CC @pganssle
There was a problem hiding this comment.
Reverted the property test to the previous deterministic checks of the first
and last days of every month around the 4-, 100-, and 400-year boundaries.
These checks use self.theclass and run without Hypothesis. The tests for the
year reported by out-of-range ordinals are retained.
Also removed the Hypothesis import from datetimetester and reverted all
changes to hypothesis_helper.py, so that file is no longer part of this PR.
Reduce duplication and make them a little more succinct.
|
I pushed a little commit to tidy up the comments. |
Hi all. Lorenz and I wish to thank you for your interest in our work. We hope this goes well and our algorithms make their way into cpython. My two cents follow.
For gcc and clang, the assembly looks good to me. Unfortunately, msvc adds a branch for this part of the code: The above is equivalent to: (see https://godbolt.org/z/PGe83cPPx). This change has no impact whatsoever on gcc. For clang the assembly slightly changes but nothing that catch my attention. Now, for msvc the branch disappears and this is a good thing. Therefore, I do recommend this change. (Please rerun benchmarks.) Finally, I have a question, out of curiosity: Why is there an implementation in Python and another one in C? I hope this helps and, again, I thank you all for considering using our algorithms in cpython. |
|
Hello @cassioneri!
Hmm, I'll think about this a bit more, but I'm still in favour of the current, clearer, logic. I hadn't seen the talk previously, nice!
These are micro-optimisations, I haven't benchmarked but I presume it will have little impact in practice. @mazalovmx are you able to test this, I'm afraid I don't have a Windows machine.
Primarily, compatibility with other implementations (e.g. PyPy uses just the Python implementation), PEP 399 has more detailed information. |
Restore deterministic ordinal boundary tests and revert the unrelated Hypothesis warning workaround. Use Cassio Neri's suggested ternaries for the January/February adjustment; MSVC emits conditional moves in the standalone function, with no consistent API speedup in local measurements.
|
Applied the suggested ternary change and tested it on Windows 11, AMD Ryzen 5 3500U, Your code-generation observation reproduces: in a standalone I also built commit
These reuse the six benchmark loops and inputs from my earlier script. Both builds passed After restoring the deterministic tests and reverting the Hypothesis helper |
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
gh-161746
Replace ordinal-to-date conversion with the Neri–Schneider algorithm
in both the C and pure-Python implementations. The C version follows
the same arithmetic steps, uses 32-bit intermediates for valid dates,
and preserves the year reported for out-of-range ordinals.
Verification of the review follow-up
10.0.26100.0, Release with LTCG, without PGO.
test_datetime,test_calendar,test_strptime,test_time,test_zoneinfo,test_plistlib,test_xmlrpc, andtest___all__:SUCCESS; run=1,615 skipped=151. The build omits optional extensions including SSL
and ctypes; skipped tests are recorded in the test log.
algorithm: zero mismatches for both C and pure Python.
arguments, and ISO upper bounds, including 20,008 large ordinals for both
dateanddatetime.400-year boundaries, as requested. The out-of-range year tests are retained.
The PR no longer changes
hypothesis_helper.py, anddatetimetesterimports with warnings treated as errors without loading Hypothesis.
against an independent Gregorian counter for all 3,652,061 internal
ordinals through
MAXORDINAL + 2.Benchmarks
Previously reported Linux measurements at b3f74a0, before the Windows
ternary follow-up. These were not rerun as part of the Windows check.
pyperf,
bench_ordinal.py, main (e891a91) vs. the patch; µs (C) or ms(pure Python) per 1000 operations, mean of the pyperf runs:
date.fromordinal, random datesdate.fromordinal, consecutive datesdate + timedelta(days=40)datetime + timedelta(days=40)date.fromisocalendardate + timedelta(days=1), 1000 stepsIn C,
date + timedelta(days=1)does not callord_to_ymd()(
normalize_y_m_d()handles the step to the next month directly), so no gainis expected there. The historical pyperf run reported about a 3% slowdown
in this control benchmark as significant; it was not rerun here.
Windows/MSVC review follow-up
The final C adjustment uses Cassio Neri's suggested conditional expressions.
MSVC 19.44.35230 x64 emits conditional moves instead of the final conditional
jump in a standalone
/O2build of the conversion function.Compared commit
8ea445bffa8fwith and without this change on Windows 11,AMD Ryzen 5 3500U, x64 Release with LTCG, without PGO. Times are microseconds
per 1,000 operations:
date.fromordinal, random datesdate + timedelta(days=40)date.fromisocalendardatetime + timedelta(days=40)date.fromordinal, consecutive datesdate + timedelta(days=1), 1000 stepsThe six original benchmark loops were run through a stdlib runner, not
pyperf: 12 paired process runs, alternating A/B order, pinned to one logical
CPU, with three warmups and five samples per benchmark per process. The
95% paired-bootstrap intervals are for baseline time divided by time with
the change; all include 1. No consistent API speedup or regression was
detected. These measurements isolate the ternary change; they do not compare
the new calendar algorithm with the old algorithm on Windows. The C source
in the final follow-up is identical to the measured ternary version.
Earlier measurements (first version of the patch, 64-bit form; superseded)
pyperf, x86-64 VM, gcc 13.3,
-O3, no PGO; C implementation, µs per 1000 operations:date.fromordinal, random datesdate + timedelta(days=40)date.fromisocalendardatetime + timedelta(days=40)date.fromordinal, consecutive datesdate + timedelta(days=1), 1000 stepsWith a PGO build (
--enable-optimizations, gcc 13.3) of main (e3262d9) and ofthat version, 1,000,000 operations, best of 7, builds alternated:
date.fromordinal, random datesdate + timedelta(days=90)date.fromisoformat('2024-W07-1')date + timedelta(days=10)pyperformance 1.14.0 (
--fast, same two PGO builds, 119 benchmarks in common):geometric mean 1.00x. No pyperformance benchmark exercises this code path, so
no change is expected there.
AI tools assisted with implementation, review, and local verification.